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Executive Summary 


This report reflects on progress from over eight years of research projects in the 
cyberlearning community. The community involved computer scientists and learning 
scientists who received NSF awards to investigate the design of more equitable learning 
experiences with emerging technology—focusing on developing the learning theories and 
technologies that are likely to become important within 5-10 years. In early 2020, our 
center's team analyzed the portfolio of past and current projects in this community, and 
convened a panel of experts to reflect on important trends and issues. 


The concept of “ambitious mashups" arose during this review. We observed that 
cyberlearning projects combine many elements in novel integrations, leading to the design 
and study of promising approaches to a range of learning goals—these projects are not 
about only one type of technology. The projects are also unmistakably ambitious. We 
observed that projects push toward frontiers of Knowledge, seek to advance design, theory, 
methods and applications simultaneously, and tackle big learning goals and not just 
incremental improvements to the status quo. Cyberlearning projects design ambitious 
mashups and they investigate what happens when learners inhabit the designs. 


Our review found five themes that account for the most dynamic elements of the 
cyberlearning community: incorporation of Al in learning, advances in learning theories, 
expansion of research methods, incorporation of out-of-school learning, and the alignment 
of scientific policy development to NSF projects and beyond. For each of these themes, the 
report elaborates the nature of each area, describes how the community changed over time, 
shares reflective questions that arose through our review, and shares recommendations from 
our expert panel. 


We conclude with three overarching recommendations from the panel. First, this field of 
research should communicate more clearly about the unique characteristics of its research— 
to distinguish this dynamic (yet relatively small) body of work from more general waves of 
educational technology R&D. Communicating clearly can encourage proposals and awards 
that offer distinctive value and sharpen the contributions this work can make to society. 
Second, the field needs to strengthen reporting of project outcomes. At the moment, it is too 
hard to learn what is being accomplished. Third, the field should continue to sharpen and 
advance the equity work within its thematic research. 


An online companion to this report offers resources to enable readers to explore 
cyberlearning research more deeply and is available at: 
https://circlcenter.org/resources/reflections-report/. 
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Preface: Thoughts from the CIRCL Team 


“Yesterday is gone. Tomorrow has not yet come. 
We have only today. Let us begin.” 


— Mother Teresa 


For eight years, our team at the Center for Innovative Research in Cyberlearning has had the 
privilege of working with a dynamic community of researchers who were awarded grants by 
the National Science Foundation in the area of cyberlearning. We've been inspired by the 
depth of concern in the community for issues of equity as technology advances to shape 
learning in more and more contexts. 


This community, we've learned, definitely does not dwell on historical technologies for 
learning or even those that are commonplace today. Technology for learning is changing 
rapidly, and by the time today’s research could do something to improve older 
technologies—for example, those typically found today’s online courses—these technologies 
will be a “yesterday” that is gone. 


The community is oriented to exploring future learning that leverages newly emerging 
technologies—technologies that are not yet here in terms of broad acceptance, but which 
likely will become important to large numbers of future learners. Advances in speech, 
gesture, and vision are examples that may be leveraged in future learning tools, but still aren't 
commonplace in today’s educational technology. 


What learning theories will we need to use these capabilities well? What new equity 
challenges will emerge and how can we tackle them? How can we work now towards 
a better future of learning, 5-10 years from now? 


To answer these questions, researchers in the cyberlearning program bring together three 
components: computer science expertise, learning sciences theories and methods, and a 
commitment to investigating equity. The core challenge of the research is to figure how we 
can begin today to study and improve a tomorrow which has not yet come. Projects involve 
designing prototypes that anticipate possible futures and studying them, especially with an 
eye to uncovering the issues and strengthening the theories needed to address those issues. 
We at CIRCL have had the role of organizing a community of such projects, sharing 
Knowledge across boundaries, and seeking synergies among the individual projects. 


In March 2020, along with many colleagues, we found ourselves working in new situations 
due to the coronavirus pandemic. Also, we Knew that the funding cycle for CIRCL would 
come to an end soon, in September 2020. We decided to make use of the disruption in our 
routines as well as the pressure of time to reflect deeply on how the portfolio of projects in 
this community evolved over eight years. We worked together to analyze the considerable 
archives we had of past projects, reports, convenings, and themes. And we invited external 
experts to join a virtual meeting to reflect with us. These experts included all the past NSF 
program officers involved in cyberlearning as well as many of CIRCL’s advisors over the 
years. 


In our reflections, we realized how the emphasis on the researchers in the portfolios evolved. 
The program didn’t proceed as its designers first imagined—well, not exactly, anyway. But it 
did proceed in ways that appear to make it more relevant and important for the future. One 
important example is that the program originally aimed to explore “genres” of future learning 
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technology, a term that eluded clear definition. Through our reflective examination of the 
archives, we observed that the program instead appeared to emphasize “ambitious 
mashups"—projects in which a set of novel technologies were brought together in 
unimagined ways to tackle a learning challenge—with researchers studying how learning 
proceeds in that mashup. By exploring a particular ambitious mashup, researchers aimed to 
gain design insights, to elaborate learning theories, and to refine research methods to work 
better with future technologies. 


We anticipate that more research like this will occur in the future. Our society faces a tension 
between the fast pace of change in technology for learning and the slow pace of developing 
research-based understanding of how to equitably use those technologies. Hence, we take 
this pause to reflect on the character of “research like this” and to seek recommendations for 
how the community can do it better in the future. What can we do better with our “today?” 


Although as researchers we operate in a different space than Mother Teresa, her quote 
resonates with us as researchers and challenges us to work now for more equitable 
educational futures. We recognize that technology is a huge part of how learning happens 
now (especially with remote learning in this pandemic era) and it’s likely to be an even bigger 
and stranger part of how learning happens in the future. We recognize that the incessant 
arrow of time may produce learning technologies that disadvantage some students or that 
make today’s inequities worse. We need to urgently anticipate the issues and conduct 
research to address them. We can't think of any other community of similarly funded 
researchers exploring future learning theories, emerging technologies, and equity challenges 
with the intensity of this research community. So, it is up to this cyberlearning community to 
do this work now, before the future arrives. We have today. Let's begin. 


The CIRCL Team 
Jeremy Roschelle, PI 
Sarita Pillai, co-PI 
Patricia Schank, co-PIl 
Kevin Brown 

Carly Chillmon 

Judi Fusco 

Shari Gardner 
Wendy Martin 
Jonathan Pittman 
Pati Ruiz 
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Introduction 


This report reflects on the progress from over eight years of research projects in the 
cyberlearning community. The projects in this NSF-funded portfolio each involved computer 
scientists and learning scientists working together with a focus on exploring the future to 
support design of more equitable learning experiences with technology. Over the past 6 
months we analyzed the portfolio of past and current projects, and met with expert advisors 
to consider important trends and issues. We begin this report by discussing a shared 
characteristic of many of the projects that became apparent to us as we examined the 
portfolio—and which resonated with the panelists who met with us to reflect on the 
portfolio. 


What are Ambitious Mashups? 


The concept of “ambitious mashups” describes what makes this collection of projects 
special and worthy of review. We first tried to categorize or tag the projects, but soon 
realized that many projects defied simple characterization. Projects did not fall into one 
major technology that they were exploring. Nor did they only refine one research method. 
Nor did they develop only one type of learning theory. The interdisciplinarity of the projects 
created new combinations of methods to address problems. Equity concerns that arose were 
also not singular. 


The projects we reviewed were mashups—they combined many elements in a novel 
integration, to explore a promising approach to a particular learning goal or context. A 
project, for example, might involve elements of artificial intelligence, virtual reality, and 
dynamic simulations. It might involve theories of collaborative learning, learning from 
representations and embodied learning. Methods could involve analyzing discourse, 
embodied cognition, and learning analytics. 


The projects also were very ambitious. These were not integrations of the most established 
aspects of each technology, theory, or method. Rather they were pushing for frontiers— 
trying to explore combinations which do not easily co-occur with today’s learning 
environments, but might co-occur in the future. Projects were also ambitious in terms of 
wanting to study big learning goals; they did not focus on incremental improvements in test 
scores. Rather they were working to open up difficult subjects so that more students could 
learn them and/or challenging who can learn important scientific topics in ways that feel 
personally relevant and connected. 


Cyberlearning projects design ambitious mashups and they investigate what happens when 
learners inhabit them. The output isn’t usually intended to be a “solution” that “scales up”— 
that would be a near-term goal to apply technology to learning. Rather the output is usually 
intended to reach a better understanding of a technology, a theory, a method, or an 
approach to equity—an understanding that might be applied more broadly in the future as it 
arrives. Indeed, cyberlearning projects also have a strong focus on groups that are 
underrepresented in STEM professions, with projects often stating a specific focus on 
underrepresented minorities, women/girls, low-performing schools, low-income settings, 
students with a disability, or students who are learning English. 


The report that follows explores the intents and outputs of cyberlearning projects in more 
detail. 
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About the Cyberlearning Community 

Our center, CIRCL, nurtured a community among the investigators of separately funded 
NSF projects that either were directly funded by one of the cyberlearning programs at NSF 
or were tagged by NSF program officers as cyberlearning-relevant. About 450 projects and 
two thousand investigators participated in at least one CIRCL event. A recent exit survey of 
cyberlearning projects found that almost all projects include both a computer scientist and 
a learning scientist. A more detailed look reveals that: 


e Prominent education or learning disciplines include Curriculum and Instruction, 
Cognitive Psychology (or Cognitive Science), Informal Education, and Special 
Education. 

e Prominent computer science areas include Artificial Intelligence, Machine Learning, 
Learning Analytics, Artificial and Virtual Reality, Natural Language Processing, and 
Biometrics 

e Projects often also have expertise in a STEM disciplines such as Life Sciences, 
Engineering, Mathematics, and Digital Media (or Communication). 

e Other cited disciplines include Social Science, Business, Humanities, Graphic Design 

e Teachers are partners in 58% of projects 

e Almost all projects train at least one graduate student, usually with multidisciplinary 
focus 


Despite the overall underrepresentation of women in Computer Science, cyberlearning 
projects have more women than men as computer science Pls of projects in recent years. 


Projects take place in a mix of settings, with strong representation of both formal and 
informal education settings, as well as a small but growing number of projects in 
workplaces. Across the portfolio, projects work with learners of all ages. Individual projects 
may be explored via the interactive Project Tag Map. 


Process for Identifying and Analyzing Themes 


Our team chose and analyzed key themes for the Reflections Briefs and expert panel meeting 
using a hybrid approach of bottom-up (data driven) and top-down (knowledge driven) 
methods. 


The bottom-up approach leveraged tags derived from the cyberlearning portfolio. All NSF- 
funded cyberlearning projects on the CIRCL website are tagged by one or more of roughly 
50 tags (see Figures 1 and 2). These tags were generated by the CIRCL portfolio analysis team 
based on content reviews of CFLT projects across the years. Proposed tags are also included 
in project intake and exit surveys sent to Pls, who are asked to indicate which tags apply to 
their projects and/or suggest new tags. Since not all Pls respond to these surveys, the CIRCL 
team reviewed project abstracts and assigned tags when they were needed. Further, in 2018, 
the team conducted a major review of the full tag set and did some consolidation and 
retagging. 
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Projects Perspectives Primers 


Project Tag Map 


A 


Resources 


Newsletter Events 


F 


Q Contact About Educators 


P 


Accessibility and technology (8) 
Al / Machine learning (19) 
Attention (1) 


Augmented/virtual/mixed reality (30) 


Cc 


Formative Assessment (10) 


G 


Games and game making (31) 


Citizen science (14) 

Collaborative and/or participatory learning (50) 
Communities for learning (19) 

Computational thinking (25) 
Computer-assisted assessments (15) 


Computer and information science (32) 


D 


Data analytics/data mining (34) 
Data science learning (20) 
Data visualization and representations (23) 


Design-based research (22) 


E 


Embodied learning (18) 
Engineering (20) 
Equity (6) 


Evidence-centered design (3) 


Identity in learning (9) 
Informal learning (27) 
Inquiry learning (26) 


Intelligent tutors and tools (37) 


L 


Language learning (9) 


M 


Makers/making (13) 

Math learning (29) 

Mobile learning (15) 
Modeling and simulation (21) 
MOOCs (5) 


Multimodal data/environments (12) 


N 


Neuroscience (2) 


Peer production (1) 
Personalized learning (15) 


Professional development (24) 


R 


Robots/robotics (10) 


s 


Science learning (42) 

Smart & connected communities (5) 

Social emotional learning (11) 

Social sciences learning (19) 

Speech technologies / Natural language processing (6) 


Statistical methods or research design (13) 


Vv 


Virtual and remote laboratories (6) 


w 


Workforce (17) 
Workshop (35) 


Writing/argumentation (9) 


Figure 1: All project tags from the Project Tag Map on the CIRCL website. 


Common Tags - Project Tag Index 


Computer and information science Games and game 
making Mat rning Data analytics/data mining 


Science learning Workshop 
caning Collaborative and/or 
participatory learning 
Augmented/virtual/mixed reality Intelligent tutors and 
tools 


Figure 2: Sample project tags from tag cloud on CIRCL website. 


For the Reflections analysis, the team counted unique tags across projects, created clusters 
of related tags and projects, discussed the counts and clustering results, and selected a set of 
themes for analysis. The themes emerged primarily from (bottom up, data driven) coding 
frequencies, but the CIRCL team's (top down) knowledge and experience doing the 
Cyberlearning Community Report also helped us think through important themes and 
meaningful clusters, influencing the process. For example, one tag that was frequently 
selected was Informal Learning, which indicated that many cyberlearning projects take place 
outside of the classroom. Our knowledge of the community helped us to understand that 
cyberlearning projects in formal education settings would likely be included in themes such 
as Learning Theories and Al & Education and would not need to be under a separate theme. 
We decided to have an Informal Learning theme that included projects focused on making, 
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citizen science, and interactive/mobile exhibits. On the advice of panelists, we changed the 
name to Out-of-School-Time, to be more accurate about the kinds of projects that were 
subsumed under this theme, since some could involve camps or afterschool programs that 
were not necessarily “informal” but which were not part of a standard school day. The theme, 
Emerging Trends, was derived from our (knowledge-driven) discussions of the themes and 
our understanding of external trends such as NSF Big Ideas, neuroscience and learning, and 
issues around equity and access. 


Once the key themes were identified (see Table of Contents for the five themes), CIRCL team 
members organized as small group “theme teams,” typically with 2 staff members per 
theme. We then assembled a large collection of relevant documents—including descriptions 
of all convening and workshop sessions and of all funded cyberlearning projects. We 
included documents written for or by CIRCL that appear on our website, such as Primers, 
Project Spotlights, and Perspectives. We also included STEM Video Showcase videos tagged 
with “cyberlearning.” We coded each piece of information by theme and subtheme. For 
example, for the Al theme, subthemes included intelligent tutoring systems; machine 
learning; speech, vision, and naturalistic interaction; and social robots and avatars. Counts 
were tallied (see Figure 3), and theme teams examined the data for thematic changes over 
time and identified illustrative examples and highlights. Some teams also contacted individual 
Pls for their reflections on changes in themes over time. 


CL Meeting Sessions - Coding Analysis xx ro) 
File Edit View Insert Format Data Tools Add-ons Help Last edit was 8 days ago 
~~ BP 100% ~ §$ % O .00 123~ — Nunito ~ 4 » BISA % BH =ri-l- Yr @HMY-~ <- 


ALL SESSION TYPES COMBINED 


A B c D E F 6 H 1 J K L M N fe) 


ALL SESSION TYPES {COMBINED 


1 
2 Cross cutting shot up in 2019. Not surprisingly, Methods, Representations, Learning Theories, and Cross cutting were most frequent overall 
3 2013 2014 2015 2016 2017 2018 2019 SUM 
40 = METHODS 
4 METHODS 2 3 6 18 «13 7 44 a 
5 |Al 0 0 0 5 3 7 15 30 / = REPRESENTA 
6 REPRESENTATIONS 1 16 8 16 06 «11 9 44 / = LEARNING TH 
7 LEARNING THEORIES 3 8 8 16 15 16] 55 * = Z a iaeOniiAL 
8 INFORMAL 4/3 3/2] 2) 43) | 4 BFS = aaaacir 
8 EMERGENT 0 1 0 5 14 10 29 CROSS CUTTI 
10 CROSS CUTTING 0 5 15 17. (15 35 82 0 
"1 7 36 40 #79 77 86 282 
12 
3 BY SESSION 
14 EEs were fairly widely distributed, but cross-cutting and emergent dominated slightly 
15 EXPERTISEEXCHANGES 2013 2014 2015 2016 2017 2018 2019 SUM 7 ei METHODS. 
‘6 METHODS 2 0 1 1 4 a 
pa“ 2 1 2 2 : 7 ~~ REPRESENTA. 
18 REPRESENTATIONS 1 1 1 0 3 3 = LEARNING TH 
19 LEARNING THEORIES 1 1 1 3 6 2 = INFORMAL 
20 INFORMAL 0 0 1 0 1 1 = EMERGENT 
21 EMERGENT 0 3 4 3 10 « CROSS CUTTI 
22, CROSS CUTTING 5 5 3 2 15 0 
23 9 11 12 11 43 
24 ~— Plenaries were mostly cross-cutting followed by learnng theories, reps & informal 
25 PLENARIES 2013 2014 2015 2016 2017 2018 2019 SUM 
26 +METHODS 1 1 0 oO 0 o} 2 . es: METHODS 
a |Al 0 0 oO 0 0 0 4 rt = 
28 REPRESENTATIONS of 5} of =| o| 6 é Cs peat 
22 LEARNING THEORIES 1 5 2 0 0 il 9 MM \ J) SS NEARING TH. 
+ & SUMMARY DATA + Al SUMMARY DATA + Methods SUMMARY DATA + Representations SUMMARY DATA + Emergent SUMMARY DATA + 


Figure 3: Analysis of themes by year and type of document or event. 
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Finally, theme teams reviewed these documents and analyzed the content therein, which 
included the removal of “Representations” as a theme since it overlapped with existing 
themes. Based upon their review, they wrote up findings and reflections on their theme in 
draft Reflection Briefs, presented these briefs in a meeting of all CIRCL staff, and then revised 
the briefs based on team questions and feedback. Next, individuals on the CIRCL team were 
assigned to review at least one other team’s brief, resulting in yet another round of revisions. 


Reflections Panel 


The resulting theme briefs were then shared with a “reflections panel” in a June 2020 
workshop. The following panelists were invited and attended two half-days of meetings 
online. In a mix of large and small groups, they reviewed the briefs, reflected on the themes, 
and deliberated on their recommendations. The CIRCL team recorded these discussions and 
took considerable notes. 


The panelists included both CIRCL’s advisory panel as well as all former and present NSF 
program officers who have led cyberlearning research programs. Members are listed below. 


e Guy-Alain Ammoussou, Bowie State University 
e Amy Baylor, National Science Foundation 

e Peter Brusilovsky, University of Pittsburgh 

e John Cherniavsky, retired from the National Science Foundation 
e Danny Edelson, BSCS Science Learning 

e Chris Hoadley, New York University 

e Sherry Hsi, Concord Consortium 

e Ken Koedinger, Carnegie-Mellon University 

e Janet Kolodner, Boston College 

e Tanya Korelsky, National Science Foundation 
e Stephanie Teasley, University of Michigan 

e Erin Walker, University of Pittsburgh 


After the expert panel, authors of each brief revised their drafts based on input from panelists 
and added a new section to their brief summarizing conclusions and recommendations from 
the panel. The final versions of the briefs are presented as the five themes of this report. 


History of the Cyberlearning Program 


At the beginning of our expert panel meeting for this reflections process, we asked panelists 
to tell the group about their experience with the Cyberlearning program and what they 
imagined it would be. We learned about their perspectives on the program at the different 
points in which they played a role in its development. Together, their comments told the 
history of how the program was created and how it evolved over time. 


Before there was a Cyberlearning program there was another collaboration between NSF's 
Directorate for Computer and Information Science and Engineering (CISE) and the 
Directorate for Education and Human Resources (EHR) called Advanced Learning 
Technologies (ALT) that ran from 2005-2009. Prior to ALT, there were other initiatives in 
learning technologies such as “Learning and Intelligent Systems” and “Collaborative Research 
in Learning Technologies (CRLT).” With the advocacy of NSF from both directorates, 
including Haym Hirsh, John Cherniavsky, Amy Baylor, Elizabeth Vanderputten, and Ken 
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Whang, and under the leadership of assistant directors Jeannette Wing (CISE) and Joan 
Ferrini-Mundy (EHR), funding was secured from Congress for what was newly referred to as 
“cyberlearning.” In 2010, a Task Force on Cyberlearning and Workforce Development was 
commissioned to gather information from researchers from a number of different fields 
about what NSF could do to create a 21st century workforce that would be 
cyberinfrastructure savvy. This taskforce produced a report! which included many 
recommendations, one of which was: 


We recommend support for research in cyberlearning. The NSF should 
devote significant resources to research and development in cyberlearning, 
exploring meaningful metrics for assessing the needs and progress of all 
learners by the learner, educator, and others, and the learning impacts of 
cyberlearning resources and opportunities. A solid body of professional 
research evidence and development work is needed before cyberlearning 
tools can be effectively implemented across the learning spectrum, 
maximizing positive impacts while minimizing unintended consequences. 


Starting in 2011, NSF created a new program specifically to fund cyberlearning research. 
Janet Kolodner was asked to lead it. "When | first heard about the program, | thought it was 
simply about technology for online education, and that didn't interest me so much,” said 
Kolodner. “But then | learned Jeanette and others had more in mind and that they wanted to 
give me a chance to make it interesting.” Kolodner drew upon her experience as faculty at 
Georgia Tech and editor of the Journal of the Learning Sciences to create a program, which 
became Cyberlearning and the Future of Learning Technologies (CFLT). A solicitation was 
created encouraging the submission of proposals for innovative technology projects that 
explored how learning theories played out in educational contexts. The program funded 
design projects that represented multiple disciplines and technologies that were more 
exploratory than many of the projects funded by other NSF programs. The program also 
looked to produce findings that would generalize beyond the particulars of the current 
exploration. In 2013, the Center for Innovative Research in Cyberlearning (CIRCL) was 
established to support this new community of interdisciplinary researchers. 


In 2014, Kolodner left NSF and Christopher Hoadley, a rotator from New York University, 
became the lead program officer for CFLT. The program continued to encourage design 
projects that focused on big ideas in learning theory. Hoadley also made an effort to bring 
together the cognitive science, computer science, and learning sciences communities, 
recommending that researchers crafting proposals for the program read the latest research 
in all of these fields as well as the design field and use that to inform their projects. "We 
needed to center on design,” Hoadley said, “not only rigor and theoretical ideas, and not just 
a kitchen sink of ideas. We needed designs that would really help us to build knowledge and 
understanding of the technological possibilities that would become available in the future." 
There was less of an emphasis on research that produced scalable, generalizable results and 
more encouragement to explore challenging ideas and advancements in designing coherent 
innovative integrations of technology, computer science and learning theory. 


In 2016, NSF came out with their ten Big Ideas for investment, which had a major influence 
on the emphasis given to many programs, including cyberlearning. The Big Ideas that had the 


1 Task Force on Cyberlearning and Workforce Development (2011). A Report of the National Science 
Foundation Advisory Committee for Cyberinfrastructure. 
https://www.nsf.gov/cise/oac/taskforces/TaskForceReport_Learning.pdf 
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most relevance for cyberlearning were INCLUDES—with its focus on systematic approaches 
to broadening participation in the STEM workforce—the Future of Work at the Human- 
Technology Frontier, Convergent Research, and Harnessing the Data Revolution for 21st 
Century Science and Engineering. Senior advisor for the Cyberlearning program, John 
Cherniavsky, was particularly interested in this last idea. "| came to the program from a long 
background in new computer science advances,” said Cherniavsky. “As | came on board, big 
data and learning analytics were areas of research that were accelerating rapidly. One of my 
main interests is in tools, and thus cyberlearning expanded to include analytic tools that 
would become important parts of the designs of future technologies for learning.” 


Also, in 2016 Hoadley finished his rotation at NSF. Because Cyberlearning is a cross- 
directorate program funded primarily and led jointly by CISE and EHR (with collaboration 
from Directorates for Social, Behavioral and Economic Sciences (SBE) and Engineering (ENG), 
the directorates decided to have co-program leads from both CISE (Tatiana (Tanya) Korelsky) 
and EHR (Amy Baylor). “The field of cyberlearning had strong foundations built on the 
fantastic ideas and contributions from Janet, Chris, John and CIRCL when Tanya and | took 
over to co-lead the program,” said Baylor. “Unfortunately, the budget had been significantly 
cut, and we focused on honing the specific, differentiated value of this program. Many 
existing programs in EHR were already funding educational technology implementation 
projects. We focused cyberlearning on exploratory projects that were risky, ambitious, and 
that innovated in both the learning and computer sciences, which was important for our 
partnerships with CISE and ENG." 


Also, during our time, the NSF Big Ideas were developed by the NSF Director Francis 
Cordova, and NSF used that opportunity to challenge the field and connect cyberlearning to 
one Big Idea in particular, “Future of Work at the Human-Technology Frontier.” These led to 
increased visibility of the program across the Foundation and facilitated a larger budget. The 
name of the program changed to Cyberlearning for Work at the Human-Technology 
Frontier. In addition, to emphasize the innovative and experimental nature of the program, 
the new iteration only funded exploratory projects, rather than supporting a range of 
exploratory, development, implementation, and scale up projects, as other programs do. “We 
really focused on exploratory work as an important nexus for computer science and learning 
scientists to come together,” Korelsky said. “We wanted to understand how technology 
affected the learning process; especially with very novel technologies. We emphasized a 
midsized project with a budget big enough for both a computer science and a learning 
sciences graduate student to be engaged together.” Even though the program continued to 
fund cyberlearning research in all areas (not just in the context of work), Korelsky also noted 
that having “work” in the program title led to fewer proposals being submitted. The ones 
however that were submitted were high quality and used innovative approaches to connect 
learning with workforce preparation through technology integration. 


Now with new NSF director Sethuraman Panchanathan replacing France Cordova, there will 
be new areas of research that he will want the directorates to pursue and those priorities will 
certainly have an impact on the Cyberlearning program. However, the program’s particular 
strength in bringing diverse disciplines together to create innovative ways to support learning 
will likely continue to be valued by the larger NSF community. As one of the original advisory 
board members, Daniel Edelson stated, “Talented experts in non-education fields often 
develop innovative ideas for improving education. Where the Cyberlearning program has 
done an exceptional job is in helping innovative newcomers to develop partnerships with 
learning scientists to develop theory-informed designs and to investigate how those designs 
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support learning.” Since the program's founding it has been able to adapt to new trends and 
priorities while maintaining its core value of building knowledge within educational programs 
informed by learning theories. This adaptability is a testament to the ingenuity of the 
community of researchers that the Cyberlearning program and CIRCL have cultivated and 
supported and will continue to support into the future. 


Theme 1: Artificial Intelligence & Learning 


Authors: Jeremy Roschelle & Patricia Schank; Contributors: Judi Fusco & Wendy 
Martin 


Definition 


Cyberlearning projects advance state of the art technologies and techniques from computer 
science, data science, robotics, and other areas. In conjunction with learning scientists and 
experts in equity and in particular subject matter or educational contexts, cyberlearning 
projects seek to advance our understanding of these technologies and techniques in the 
context of human learning challenges. In reviewing the portfolio, the most common types of 
technologies and techniques are: 


e Intelligent Tutoring Systems, in which a computational agent supports and guides 
students as they work to learn challenging subject matter. 

e Machine Learning, where statistical techniques are applied to infer patterns in large 
data sets about learner behavior and outcomes. Insights and techniques for guiding 
learner's behavior to desired outcomes are sought. 

e Speech, Vision, and Natural Interaction, where machine learning and other 
techniques are specifically applied to analyze human speech, visual scenes in which 
learners participate, eye-gaze, etc. and (in cases) to synthesize naturalistic interactions 
with learners in the same modalities. 

e Social Robotics and Avatars, where above techniques are applied to provide an 
artificial learning partner to a student or group of students. 
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Invention Coach Main Interface. The goal of the task is to invent an index of “clown crowdedness” for each of the buses int! 
receive support from the Coach (B) in the dialogue box (C), and input their ideas into index generation spaces (D). Additional 
the calculator (F), taking notes in the notepad (G), reviewing the rules (H), or soliciting help from the coach by clicking “Help!” 
Virtual Peers — Used with Invention Coach ITS — Used with 
permission of Justine Cassell permission of Catherine Chase 
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Change over Time 


The Artificial Intelligence (Al) & Education community is long-standing and accomplished, 
with its own society, journals, and multiple funding streams. As participants in the 
cyberlearning community, experts in the above topics were somewhat less present in the first 
3-4 years, but participation increased strongly thereafter. Rather than simply advancing 
Intelligent Tutoring Systems (ITS) technology with Al, cyberlearning projects combine Al with 
learning theories to create “ambitious mashups.” Examples include exploring hybrid systems 
in which people and Al agents work together, focusing on social learning (classic Al & Ed 
research was somewhat more individual-oriented. Cyberlearning Al projects also support 
teachers and expand to applications like social robots for learning Chinese, Al supports for 
workforce learning, and an invention coach (whereas early ITS applications were often 
oriented to more bounded problem solving in math and science). Thus, the portfolio of 
cyberlearning projects push beyond Al beyond the boundaries of a conventional ITS. 


Another important body of cyberlearning research is not framed relative to an ITS or a “tutor” 
but rather to automated analysis of the complex human activity of learning. Understanding 
speech patterns in collaboration is one example. Automatically analyzing videos of preschool 
teachers and children is another. Assessment applications also were pursued, where the 
analysis of naturalistic input can inform us as to what a student Knows and can do. A need for 
relevant corpus of data was often discussed. 


An emergent and important initiative in the community has been to propose open platforms 
that could support multiple investigators in more efficiently experimenting with Al, Machine 
Learning, and Natural Language techniques (rather than each investigator building their own 
soup-to-nuts learning environment). ASSISTments is a platform example that was talked 
about early and often in Cyberlearning convenings, as was LearnLab from CMU. Other 
potential platforms could include science “workbench” type learning environments, shared 
robots platforms, MOOCS, large scale curricular resources (like AlgebraNation) and also 
museum experiences (like Connected Worlds) that are instrumented to provide data to 
multiple researchers. 


At the 2018 joint meeting of ICLS, Al & Ed, Learning at Scale, and the Cyberlearning 2019 
convening, issues of fairness, privacy, accountability, transparency, and equity came up very 
strongly as issues the cyberlearning community wants to work toward resolving. A 2020 NSF 
call for large Al Institutes drew much participation from cyberlearning researchers. 


Reflection Starters 


Questions posed to a panel of CIRCL advisors and current and former NSF Cyberlearning 
Program Officers for discussion: 


e Al & Education has been a long-standing field with multiple funding streams, of which 
cyberlearning funding is a relatively small amount. Is there potential for unique 
contributions from cyberlearning? How could this community advance Al in Learning 
beyond what happens in other work streams? 

e Educational settings can present important fundamental challenges for speech, vision, 
and naturalistic interaction; how could cyberlearning’s potential contribution be 
strengthened? 
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e This area is advancing very rapidly and is very technically demanding; what do these 
projects tell us about how other areas of cyberlearning can include cutting edge CS 
expertise? 

e What issues related to Al and fairness, accountability, transparency, and equity need to 
be considered in the future? Is this the place for it? Do solicitations need to 
specifically call for this? 


Recommendations from the Reflections Panel 


Cyberlearning researchers are in a unique position to encourage and enable more human- 
centered Al and hybrid Al systems—for example, with teachers in the loop across all phases 
of system development and use. Recommendations from the panel included: 


e Empower and involve teachers (in addition to subject matter experts, learning 
scientists, and technologists) from the beginning as co-designers of Al-based 
intelligent systems. 

e Enable teachers to train and correct the system as it responds incorrectly or 
inappropriately. 

e Teachers and students should also be given agency to interpret and potentially 
override a system's recommendations. 

e Projects should identify meaningful, inclusive partnerships at the research level with 
practitioners and institutions that serve underrepresented minorities during the 
proposal-writing phase. 

e An NSF-funded Al & Education resource center should help projects by brokering 
connections to broaden participation and to help connect learning research with 
new fundamental capabilities in artificial intelligence, which is a rapidly evolving field. 


Project Examples and Resources 


For project examples, resources, and links, visit: https://circlcenter.org/resources/reflections- 
report/ai/. From this site, you will be able to review project abstracts, read related 
publications, and view videos produced by the projects. 


Ambitious Mashups 10 


Theme 2: Learning Theories 


Authors: Judi Fusco & Jeremy Roschelle; Contributor: Patricia Schank 


Definition 


Theories of how people learn are an integral part of cyberlearning research. Projects 
interweave learning theories with emerging technology and research methods to uncover 
processes of how learning unfolds with theoretical depth and empirical precision in service 
of new theories of learning. Cyberlearning projects also use principles of how people learn to 
support learning in STEM (for example astronomy, data science, biology), topics beyond 
STEM (language learning, reading, writing) and workforce skills. In this review, the following 
three areas in learning theories were considered: 


e Collaborative Learning. Collaborative learning examines how people learn socially 
through shared words, actions, and meanings. In cyberlearning, collaborative learning 
occurs when the tool or environment for learning is social. The concept of 
collaborative learning is broad and includes collective inquiry, Knowledge building, 
joint problem solving, intersubjectivity, shared/collective/group/distributed cognition, 
collective consciousness, and transactive discourse. From a constructivist perspective, 
learning occurs as participants make sense of their experience. 

e Embodied Learning. Embodied learning theories seek to understand how 
movements, gestures, and actions influence learning. In cyberlearning, embodied 
design guides the integration of gesture and technology; for example, virtual tutors 
gesture in interactions with learners and technologies recognize movements that are 
used in relation to concepts. 

e Identity. Sociocultural theories examine how a person forms their own identity 
situationally, and how learning is about developing identity (not just knowledge or 
skill). Identity development includes seeking to understand how you think of yourself, 
your perception of how others see you, and what perspective is promoted by society. 


Among all themes examined in our reflection analysis, “learning theories” was the second 
most frequent tag for cyberlearning convening sessions. While here we focus on the three 
sub-areas above, other long-standing scientific theories also have a strong place in 
cyberlearning work. For example, cognitive theories are not explicitly included here, although 
cognitive theories often inform projects with respect to representations (e.g., games, 
simulations, and visualizations), methods, and Al (e.g., learning analytics and intelligent 
tutoring systems), and collaborative learning. Projects guided by neuroscience theories are 
discussed in the External Trends brief, but also overlap with projects tagged with embodied 
learning. 
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Co-constructing solutions — Gesture Augmented Simulations for 


Used with permission of Emma Supporting Explanations (GRASP) — 
Mercier Used with permission of Robb Lindgren 


Change over Time 


Collaborative Learning was seen in 36 funded projects, addressing a diverse set of topics 
including intelligent tutoring systems, learning analytics, robots to help teach collaboration 
skills, and augmenting work in maker spaces, workplaces, and citizen science. In many of 
these projects, the innovation comes from leveraging collaborative learning in new ways with 
new technologies. In other projects, the innovation has come through deepening a theory of 
collaboration—for example, developing the concept of orchestration or advancing 
knowledge building theory (e.g., in Jianwei Zhang's Thread Mapper and Tom Moher's 
instrumented classroom). In orchestration work, advances have supported ensembles of 
people taking on different roles rather than one person directing tasks. Other projects have 
advanced our understanding around how to connect people and ideas across settings (Erin 
Walker's ubiquitous collaboration technology), or how people and Al—such as conversational 
agents and robots—can work together. Technologies for supporting advances in 
collaborative interfaces include immersive experiences, such as AR and VR; voice-driven 
systems; tangible computing; tabletop devices; language processing; collaborative sketching; 
and digital ink. 


Embodied Learning was seen in 15 funded projects. As technologies such as VR, AR, and 
haptic feedback have become more sophisticated and allow the tracking of learner's 
movement, many “ambitious mashups" guided by learning theories have emerged and 
evolved over the years. Early projects sometimes involved learners moving objects on iPads, 
or experimented with wearable technologies. More recent projects took advantage of 
emerging technologies to analyze movement in more sophisticated ways (using methods 
informed by learning analytics) and provide feedback to learners (through an ITS or virtual 
tutor). One recent project gives a “minds-on" bridge between ideas and writing using 
motion-tracking, sketch-recognition, and animation technologies (see Digitally-Augmented 
Enactment). Settings for embodied projects range from helping improve reading for ELL 
students, computational thinking, mathematics, and science, and to enhance remote 
learning; participants ranged from young children to adults in the workforce. 


Theories of Identity were strongly addressed in 9 projects, often focusing on how 
participants’ identities changed through the project work. For some learners, their identity is 
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very different in home and school situations; many projects investigated how to connect 
their learning across contexts. Additionally, games and virtual coaches have been used to 
help shift habits, dispositions, and behaviors in learners. For example, the Developing STEM 
Identities project investigated the intersection of technology, pedagogy, and learning by 
creating new classroom activities to foster learning for students and to help instructors 
discover new approaches and support their students. Generally, the work done in identity has 
illustrated the broader context that is important to consider to understand and influence 
learning. Cyberlearning has also funded capacity building work to bridge sociocultural 
theories and bring them to the fore in cyberlearning work. Distinct work in equity—related to 
identity—is being done in work on Smart and Connected Communities for Learning, and 
interest-driven learning (e.g., FUSE). 


Reflection Starters 


e In what ways has cyberlearning pushed learning theory forward? How could or should 
cyberlearning research seek to develop learning theory in the future? 

e Dowe now better understand the "zone of proximal development” where emerging 
technology, learning theory, and equity each have a powerful and mutually 
informative voice? Can we make a stronger case for why research on learning 
technologies that are 5-10 years out should include research that develops, refines, 
and/ or clarifies new theory? 

e The body of cyberlearning work is theoretically diverse. Does it make sense for the 
conversation to spend more time on a few constructs (like those above) or should 
there be a more pluralistic way of working with the theoretical diversity? How do we 
choose what gets the “floor” in community events? 


Recommendations from the Reflections Panel 


Panelists described theory as highly important in cyberlearning research, to guide design, to 
enable analysis of learning processes, and to work towards generalization. Panelists 
recommended further clarification of the nature of the interdisciplinary theory development 
work that occurs in exploratory, future-oriented research, and the potential outputs of this 
work. 


e Articulating and explaining the most common important theories and stances in 
cyberlearning research will strengthen the field and help it to communicate. These 
include the areas of theory above as well as additional ones (e.g. panelists noted 
cognitive and human-computer interaction theories are prominent in the work) 

e This research community has notable opportunities to refine theories in relationship 
to design of novel learning environments and how designs can be implemented in 
varied education settings. 

e The research community has a strong opportunity to use theory in its explorations to 
generate stronger hypotheses about what can make learning more equitable in the 
future. 

e Itis important to see theory-building as convergent work of interdisciplinary 
research teams, rather than only the work of learning scientists or just one PI. 

e Exploring innovative learning tools and new forms of data (e.g. multimodal analytics) 
may open up or transform potential research questions, and thus influence many 
more researchers. 
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Ultimately, panelists were wary about the tendencies of the overall educational technology 
field to promote platforms and tools with weak or absent theory. Cyberlearning research has 
an important opportunity to further develop learning theories so they are highly relevant to 
the design of future learning technologies and also are rigorously grounded in larger bodies 
of theory and in empirical data. 


Project Examples and Resources 


For project examples, resources, and links, visit: https://circlcenter.org/resources/reflections- 
report/learning-theories/. From this site, you will be able to review project abstracts, read 
related publications, and view videos produced by the projects. 
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Theme 3: Research Methods 


Authors: Judi Fusco & Patricia Schank; Contributor: Jeremy Roschelle 


“| see exciting possibilities in cyberlearning for thinking about tool design 
and analytics design together, such that learning environments produce 
useful data and are designed to take advantage of the data to support 
teachers and students as well as their own continual self-improvement.” 


- Alyssa Wise (excerpt from 6/1/20 reflections on her work and field) 


Definition 


Cyberlearning projects explore the frontiers of learning with technology, and to do so, the 
methods they use need to be innovative and informative. While methods are part of every 
project, they sometimes become the prime focus of the project—for example, when the 
methods are uniquely innovative or being refined as part of the work. Even established 
methods can take center stage when they deeply shape the work being done. Because of the 
interdisciplinarity of cyberlearning projects, we also see methods from different fields 
combined in new ways to provide insights to questions. In reviewing the portfolio, the most 
common or growing methodological approaches found were: 


Design-Based Research (DBR). Researchers create learning experiences and study 
them to investigate potential advances and to better understand what the target users 
and communities need. In cyberlearning, DBR is a methodology that rigorously 
explores which design features have the most potential to improve learning. 

Learning Analytics. Learning analytics can provide insights to the learning process 
and are often linked to formative assessment, sometimes in Intelligent Tutoring 
Systems (ITS), to provide students with direct feedback and/or give teachers more 
information to help their students. In addition, learning analytics have been used to 
support group problem solving by helping learners acquire content knowledge or to 
support collaborative processes. 

Multimodal Analytics. Building on learning analytics, multimodal analytics is a 
method that involves the integration and analysis of multiple data sources (e.g., visual, 
audio, gestural, movement, eye gaze, heart rate, and/or other types) to help 
researchers measure learning in new ways and gain insight into learners’ abilities, 
emotions, needs, and preferences and how these impact learning interventions. 


Multimodal sensors — Used with 


Inq-Blotter dashboard — Used 


permission of Robb Lindgren with permission of Janice Gobert 
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Change over Time 


DBR was an important part of 22 funded projects and a focus of 4 Cyberlearning sessions 
across the years. Relative to its prevalence in project work, it was highlighted less in 
convening sessions, possibly because it is a well-developed method that many learning 
scientists use to understand engagement and learning in a project. In the 2017 Cyberlearning 
Community Report, a chapter was devoted to new work bringing together Human- 
Computer Interaction (HCI) methods and DBR methods to 1) better involve stakeholders, 2) 
help maintain a focus on equity, and 3) explore tensions between making technology easy to 
use and supporting the learning process. 


Learning Analytics was found to be part of 34 projects and 14 Cyberlearning convening 
sessions over the years. More established than multimodal analytics, less established than 
DBR, and rapidly evolving (e.g., with Big Data and Al technology), this method was commonly 
highlighted in both project work and convenings. The term itself has also evolved over the 
years; “learning analytics” is commonly used in the cyberlearning community, but work in this 
category is sometimes called “educational data mining” or “big data” (especially in earlier 
years)—and in 2018 and 2019, the term Multimodal Learning Analytics (MLA) came into 
common usage. Work in the learning analytics category is often guided by cognitive theories 
of how people learn; however, some projects are exploring non-cognitive factors (e.g., 
Developing Community & Capacity to Measure Noncognitive Factors in Digital Learning 
Environments). At the 2018 “Festival of Learning” conferences in London, learning scientists, 
Al and Ed researchers, and computer scientists who study “learning at scale” argued that 
learning analytics still needs much stronger connections to learning theories and to rapidly 
evolving learning environments; in many cases, applications of analytics address learning but 
use readily observable characteristics of learning without a learning theory. Recently, one of 
the nine Cyberlearning workshops on Principles for the Design of Digital STEM Learning 
Environments focused on Research Priorities in Learning Analytics. The workshop brought 
together scholars with expertise not yet connected with learning research, including data 
scientists who have made contributions in other parts of the economy, with members of the 
business community who use “people analytics” and other methods for looking at 
competence. 


Multimodal Analytics was incorporated in roughly a dozen funded projects and five 
Cyberlearning convenings sessions across the years. Relative to its prevalence in project 
work, it was highlighted more in convening sessions, probably because it is a new, evolving 
method that many participants wanted to learn about. In the Community Report, it was 
featured as an innovative method that integrates multiple data types, including those related 
to emotion and movement, to help researchers better understand learning in multimodal and 
embodied learning environments. Recently, this was the focus of a report from a workshop, 
which noted that while multimodal sensing holds promise for better understanding learning 
and motivation, there are technical challenges to such investigation as well as the potential 
to disrupt the learning context altogether due to the intrusive presence of the sensors and 
instruments. In addition, there are challenges to understanding what students are thinking 
from the instrument-collected and automatically analyzed data for Knowing when and how 
to intervene. 
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Reflection Starters 


e What investigations do these methodologies enable or help us to advance in 
cyberlearning? Pushing the boundaries of methods for studying learning has not been 
prominent in CL solicitations; should the community push for more visibility of the 
importance of advancing methods so that we can better study learning in future 
learning environments? 

e What methods really fit a community that is exploring “ambitious mashups"? What are 
the missed opportunities or latent possibilities to further develop these combinations? 

e Learning analytics and educational data mining have societies, journals, conferences— 
what is unique about the cyberlearning community as a place for methods 
development? 

e Designing for better learning data. How can we accelerate the move from easily 
available data to designing systems that can generate more meaningful data to 
understand learning? Given “ambitious mashups” in methods as well as in learning 
environment design, is this what “convergence research” (an NSF Big Idea) should 
mean to the cyberlearning community? (e.g., can we define convergence research in 
this area in terms of the need to advance methods)? 


Recommendations from the Reflections Panel 


The discussion ranged from “what methods are important in cyberlearning” to “What makes 
something a method". The panel suggested thinking about how technology can lead to 
advances in: 1) methodologies required for research in learning, and 2) methodologies for 
design and implementation. The panel noted that though computational methods were used 
frequently, projects that use sociocultural and sociopolitical methods should not be 
overlooked. Additionally, they noted there may be a slight shift towards doing research in 
cyberlearning on shared platforms, and that these could offer data and design infrastructure. 
Some key conclusions and recommendations from the panel were: 


e Astrength of cyberlearning is the cross pollination from multiple fields, such as 
education, computer science, HCl, and practice. Convergent science that also cross- 
fertilizes methods can be very productive and lead to new insights. 

e The field should continue to triangulate multiple, mixed methods (computational, 
quantitative, qualitative, multimodal, etc. analyses) to increase validity in the 
investigation of learning through the convergence of information across different 
learners, contexts, and data. 

e People think what comes out of computational analyses is objective, when it’s 
not. Humans create the algorithms and collect the data, and humans have bias. As 
Alyssa Wise (CFLT PI) notes in her reflection, “The field needs to be explicit about the 
roles of human decision making in data mining processes.” 


In generalizing models of learning, we need to be careful: Models are built based on data and 
methods used with a particular population of learners. Assuming that a model generalizes 
to other populations could perpetuate inequities even if we are trying to broaden 
participation. 
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Project Examples and Resources 


For project examples, resources, and links, visit: https://circlcenter.org/resources/reflections- 
report/methods/. From this site, you will be able to review project abstracts, read related 
publications, and view videos produced by the projects. 
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Theme 4: Out-of-school-time (OST) Learning 
Author: Wendy Martin; Contributor: Patricia Schank 


Definition 


Many cyberlearning researchers design projects for learning contexts outside of formal 
schooling, such as museums, makerspaces, libraries, and in afterschool programs that may 
be located in schools or other locations. These settings are often more open than formal 
settings to the integration and exploration of innovative technologies and learning theories. 
Out-of-school-time (OST) learning experiences tend to be playful, interest-driven, and 
intrinsically motivated, allowing for engagement in authentic practices that can lead to 
social-emotional as well as content and skills learning. OST learning formats can sometimes 
be cross-context, multigenerational, and community oriented. Cyberlearning projects in OST 
environments have used a range of technologies for multiple purposes such as data 
collection, artifact creation, documentation, and sharing/collaboration. Cyberlearning 
research in OST settings has built knowledge about how youth develop STEM identities 
through engagement with STEM content and practitioners across settings, about how 
communities co-construct meaning from data, and how youth can develop persistence and 
agency through iterative design. Some typical types of cyberlearning informal projects 
include: 


e Making, which involves creative, interest-driven tinkering and building with a wide 
range of materials from high tech (3D printers, laser cutters, Arduinos) and low tech 
(hammers, wood, fabric, glue, paper). These often, but not always, take place in 
makerspaces, dedicated facilities with a variety of materials and tools available to 
produce artifacts. 

e Citizen Science, which has learners in the community collecting data about some 
aspect of the community or environment and sharing it with researchers and peer 
citizen scientists. Sometimes learners engage in data analysis as well as developing 
the research questions so the data can be used to achieve a community goal. 

e Interactive/Mobile Exhibits, which can take place in a variety of settings, though most 
often in museums or cultural centers, but also community organizations, parks, and 
other gathering places. These exhibits draw on technologies such as augmented 
reality (AR), virtual reality (VR), mobile computing, or touch screens to create 
immersive and/or multi-sensory experiences. 
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Flower garden (left) and white dove (right) from Paper Mechatronics- Used with 
permission of HyunJoo Oh 


Change over time 


In the first years of cyberlearning, the projects for OST education settings varied widely, from 
serious games to social robots, digital badges, and wearable computers to making, citizen 
science, and innovative exhibits. The learning theories they were based on tended to be 
around identity development, motivation, agency and collaboration. When NSF came out 
with its 10 Big Ideas, this eclectic group of projects appeared to shift into a more coherent 
set of learning experiences with specific underlying purposes. Big ideas of particular 
relevance were Harnessing the Data Revolution, NSF INCLUDES, Work at the Human 
Technology Frontier, and Growing Convergence Research—as well as the Smart and 
Connected Communities effort. These ideas provided guidance to the community to place 
an emphasis on using data analytics, broadening participation, connecting to workforce 
opportunities, creating innovative partnerships, and designing cross-context learning 
experiences. 


Cyberlearning projects that focused on making transitioned from understanding how to 
support and measure learning in makerspaces to creating maker environments for a purpose, 
like social justice, or reaching particular communities, as well as using making in hybrid 
formal and informal environments. Some relevant projects include Paper Mechatronics, 
which started with an innovative way to combine high and low tech materials to provide 
creative, low-cost tinkering experiences and evolved to have a sharper focus on supporting 
engineering education for participants. In addition, early efforts such as Learning in the 
Making focused on understanding how to design and evaluate makerspaces, while newer 
projects are more “ambitious mashups"—such as Mentoring through Embodied 
Communications, which uses social, remote robots to support maker-based career and 
technical education (CTE) programming for low-income youth in US-Mexico border towns, 
and the Instrumented Learning Spaces workshop, which used innovative data collection 
tools to gather data on participant experiences in makerspaces. 


Citizen science has been a consistent approach in cyberlearning, but now projects do much 
more than just collect and share data: they use technologies to enhance mobility and data 
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collection across environments, and leverage backend data for modeling datasets or learning 
about the users. The transformation of the WeatherBlur project is a good example. This 
project began by using technology to create a non-hierarchical community of learners 
gathering climate and other environmental data in Maine and Alaska to support those local 
communities. Recent funding has been used to make it a platform that embeds more 
backend data collection and analysis to support the user experience, build models of users’ 
skills and beliefs, and respond to their needs. 


Changes in projects related to interactive or mobile exhibits can be seen in the evolution of 
the Connected Worlds exhibit at New York Hall of Science (NYSCI). This is a large-scale 
exhibit that has visitors interact with wall-size touch screens of fanciful animal and plant life 
to learn about the interconnectedness of ecosystems. NYSCI received support to hold a 
convening about how to use the data from tech-infused exhibits to conduct research on 
visitor experiences, and then received a grant to add this form of data analytics to their 
existing exhibit. NYSCI also worked with University of Washington on Mobile City Science to 
combine aspects of the virtual museum exhibit experience and citizen science by having 
youth learn about and collect data on their communities through mobile computing and 
augmented reality. 


The WeatherBlur project — Used with Connected Worlds exhibit — Used 
permission of Ruth Kermish-Allen with permission of Stephen Uzzo 


Reflection starters 


e Given NSF's and the cyberlearning community's growing interest in leveraging data 
analytics for learning, how does the community address data privacy and other ethical 
concerns related to using participant data from tech-supported learning across 
multiple contexts/environments? What are the key concerns and challenges? How do 
OST contexts shape these conversations in a way that is different from those within 
the traditional school day? 


e Considering NSF's focused interest in work at the human-technology frontier, how 
can approaches such as citizen science and making be used creatively to support 
development of the workforce of the future, especially in underserved populations? 


e What new kinds of “ambitious mashups” of technologies and OST approaches can 
cyberlearning researchers conceptualize to broaden participation in STEM? How can 
we create experiences to inspire and support these innovative collaborations? 
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Recommendations from the Reflections Panel 


The panelists observed that the focus of Cyberlearning research on OST learning should not 
be on the setting or location or even the technologies that are used but on the kind of 
learning that is supported. In particular, these kinds of projects often examine learner agency 
and identity development in ways that can be informative in designing programs for a wide 
range of educational settings, which may include informal experiences that take place in 
schools as well as other places. Recommendations for future work in this area include: 


e Design technology-supported hybrid in-school and OST experiences, building on 
what we have learned about authentic and interest-driven learning experienced in 
OST settings to create more engaging in-school STEM instruction that enables 
learners to have agency. 


e Conceptualize citizen science projects and mobile exhibits/experiences more broadly 
as representations of community learning, in which learning is a collective and the 
community members help each other learn. 


e Capitalize on this time of remote learning, when there is a blurring of where formal 
schooling ends and OST learning begins to design technological supports to blend 
the two more intentionally and productively to make formal learning more self- 
determined. 


e Build on the affordances of digital games and play in general to motivate 
engagement and immersion in alternate realities to achieve learning goals. 


e To address ethical issues related to concerns about collecting data through 
technologies used in OST settings, draw on the lessons of citizen science and involve 
learners in deciding what data should be collected and for what purposes, as well 
as in the analysis of those data. 


Project Examples and Resources 


For project examples, resources, and links, visit: https://circlcenter.org/resources/reflections- 
report/out-of-school-learning/. From this site, you will be able to review project abstracts, 
read related publications, and view videos produced by the projects. 
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Author: Sarita Pillai; Contributors: Jeremy Roschelle & Judi Fusco 


Cyberlearning projects have been conceptualized and implemented in the midst of a 
changing educational, policy, and funding landscape. A significant influence on this 
community has been the evolution of NSF's mid- and long-term research and investment 
priorities. In 2017, some of these priorities coalesced in the agency's 10 Big Ideas. Among the 
Big Ideas, three have already had notable influence in cyberlearning projects: 


1. Harnessing the Data Revolution. Engaging NSF's research community in the pursuit 
of fundamental research in data science and engineering, the development of a 
cohesive, federated, national-scale approach to research data infrastructure, and the 
development of a 21st century data-capable workforce. 

2. Convergence Research. Merging ideas, approaches, tools, and technologies from 
widely diverse fields of science and engineering to stimulate discovery and innovation. 

3. Future of Work at the Human-Technology Frontier. Catalyzing interdisciplinary 
science and engineering research to understand and build the human-technology 
relationship; design new technologies to augment human performance; illuminate 
the emerging socio-technological landscape; and foster lifelong and pervasive 
learning with technology. 


A fourth Big Idea, INCLUDES, forms alliances to transform education and career pathways to 
help broaden participation in science and engineering. And more recently, NSF 2026 which 
seeks community input into foundational research areas of the future that are boundary- 
crossing and out of the programmatic ‘box’ and that require long-term commitment at a 
significant scale. 


NSF’‘S 
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Change over Time 


In reviewing the portfolio, we noted four significant trends/themes have a policy resonance. 
Some map to the aforementioned 10 Big Ideas; others have other resonances to other 
initiatives at NSF or beyond. 


A first theme, Data Science Learning, teaches learners the tools, techniques, and habits of 
mind involved in experimentation, data collection, and analysis—and relates to the 
Harnessing Big Data idea. In the era of Big Data and a world increasingly awash in massive 
data sets, greater attention is being paid to the importance of building the foundational data 
science skills starting in K-12 education. The Data Science Learning theme was present in 
very early cyberlearning community activities, and one important platform, CODAP, was 
funded by an early grant. CODAP is a platform designed explicitly for data science education, 
allowing students to dynamically explore relationships, build visual representations, and to 
create and modify hierarchical structures, an important part of experiencing the doing of 
data science. CIRCL convenings provided opportunities for community organizing around 
the theme, although eventually, gatherings took on their own life outside of official 
cyberlearning events. The community has also discussed important issues related to the 
goals of data science education, the use of big data in educational research, theory-informed 
use of big data, the relationship between big data and convergence research, the barriers to 
integrating data science into education, and the pervasive issue of data ethics and privacy. 


A second theme, Neuroscience, has been a focus of interest in the community and 
convenings have featured keynotes, expertise exchanges, and other professional activities on 
the topic. In 2013, the Obama administration launched a multi-agency Brain Initiative (and it 
continues today). Education researchers and neuroscientists are studying how neuroscience 
can influence learning, teaching, and the design of learning environments, and, in turn, how 
learning and teaching practices can inform neuroscience models. Although neuroscience is 
not one of the 10 Big Ideas, the convergence of cyberlearning, educational research, and 
neuroscience began early in the life of the cyberlearning program, with researchers exploring 
ways in which advances in technology can bridge the learning sciences and neuroscience In 
impactful ways. Despite a level of ‘experimental skepticism’ on the part of both technologists 
and neuroscientists, projects on multimodal models of learning, embodied learning 
experiences, virtual reality, and gaming have explored how neurocognitive tools such as 
fMRI, eye tracking, EEG, and fNIRS can be used to advance cyberlearning research and 
development. Increasingly there is interest in the needs of neurodiverse learners and those 
who learn ‘differently’ from the majority such as in the SAIL project. Several cyberlearning 
projects have focused on the application of neuroscience to inform interventions that target 
specific populations of students but many experts in the community posit that designing for 
diversity benefits all learners. 


About halfway through the life of CIRCL, one of our program officers became involved in the 
NSF Smart and Connected Communities program and asked CIRCL to host an IdeasLab on 
the topic. The third theme, Smart and Connected Communities, utilize networks and 
technology to foster lifelong learning and problem solving in formal and informal settings, 
neighborhood communities, smart cities, across countries and the world. The IdeasLab was 
successful and led to several new collaborations among Pls. Cyberlearning published about 
“Smart and Connected Communities” and hosted further events at convenings over time. 
The Partnerships for Urban STEM Learning Hubs brought together educators, researchers, 
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technologists, and students across multiple organizations within a city to develop and test a 
community learning hub design to support skill, literacy, and agency development for 
underserved youth. Complex issues remain to be grappled with, such as unanticipated 
conditions and changes that cause ripple effects across an interconnected system, 
establishing trusting relationships between partner entities and learning participants, issues of 
equity and access particularly in urban or rural locations, and data privacy concerns. 
Nonetheless, this hasn't been a recurrent theme of many projects. This could be because 
issues of learning are not highly featured in mainstream Smart and Connected Communities 
projects—and because cyberlearning projects that explore related issues — like involving 
students in mapping their communities have an adequate theoretical language without 
“smart and connected.” 


The Future of Work theme emerged last among these and has gathered momentum in 
directly funded projects. The field has envisioned a new machine age where humans will 
shape technology, where technology will shape human interaction, and where technologies 
and humans will collaborate in new ways to discover and innovate. Society's response to the 
COVID-19 pandemic is pushing us more rapidly; the future of work is likely to look much 
different than was conceived of even three months ago as more and more organizations 
facilitate learning in order to continuously transform themselves. While there has been 
limited attention to learning in the context of work within the cyberlearning community in 
the past, recent projects have begun to explore how new forms of work that engage teams 
with technologies can also provide stronger opportunities to learn and grow. There clearly 
remains work to be done to help more current and potential Pls understand what important, 
exciting projects “at the frontier” of work can look like and the role cyberlearning plays in 
transforming future workplaces into learning organizations. 


The INCLUDES initiative has been highly relevant to cyberlearning which is inherently about 
design to support all learners. Many current INCLUDES projects are “pilots” that organize a 
group of stakeholders to tackle a challenge in educational and career pathways and few have 
a learning technology emphasis. 


With NSF 2026 and the related Mid-Scale Research Infrastructure program, there is an 
opportunity for the cyberlearning community to think about its role in the future of learning 
and knowledge building. There remains a missing infrastructure component to the 
cyberlearning program that, if it was to be put into place, could enable outcomes to be more 
readily scaled. While the earliest version of the program attempted a tiered development 
category approach with an eye towards scaling impact, it never quite achieved that potential. 


Reflection Starters 


The CIRCL team has facilitated the presence of themes in the cyberlearning community that 
come from NSF policy or from external trends, and that have the possibility to resonate in 
this community's research. We see from the above cases that these themes take different 
trajectories: some germinate and then come to fruition elsewhere; others grow slowly; 
others gather some but perhaps limited momentum. 


e How should we think about how thematic research (like cyberlearning) should relate 
to broader initiatives (like Big Ideas)? 


e What can we learn from how such intersections have played out already? While 
connecting to broader initiatives involves considerable work, can we begin to 
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formulate case scenarios and guidelines for choosing and engaging with broader 
initiatives? 


Recommendations from the Reflections Panel 


The panel reviewed the past intersections and recognized the value in each of them -- but 

also that the rationale for pursuing these particular intersections should be continuously re- 
evaluated as policy leadership and policy priorities shift. Overall, the panel focused on three 
questions: 


1. What does it mean to “scale up" in cyberlearning? 

2. What mid-scale infrastructure would most benefit this community? 

3. Where does the amassed knowledge within the cyberlearning community go 
next? 


The panelists recognized that funding streams already exist at NSF and elsewhere for the 
typical sense of “scaling up.” In this typical sense, a team might take results from an early 
exploration of an ambitious mashup under cyberlearning and seek funds elsewhere to more 
fully develop a program or product. A different sense of scale up is the ability to scale up this 
community's explorations of a particular design challenge; such that multiple teams and 
expertise might contribute to more rapidly and deeply advancing an exploration. One 
panelist wondered what it would be like if multiple projects could work together on aspects 
of “how do families engage in future learning with advanced technology?” or “how 
trajectories of learning data science could be supported across the lifespan.” This led to the 
consideration of two intersectional ideas which, to date, have not been mined by the 
cyberlearning community in depth. 


e Panelists recommended more intensive consideration of mid-scale infrastructures, 
which could be platforms, data sets, or instrumentation that enable multiple research 
projects to more rapidly conduct related cyberlearning explorations. The CIRCL team 
noted, for example, that one recurrent issue is that efforts to explore natural language 
interfaces in cyberlearning frequently bring up the need for the community to 
develop a more relevant corpus of social learning speech on a common platform -- a 
problem that is bigger than any one project. Instrumentation for multimodal analytics 
is another very challenging mid-scale problem that frequently arises in cyberlearning 
projects. A toolkit or platform to support ethical and equitable Al in learning research 
is also “infrastructural” in the sense of supporting many projects. 


e Panelists also recommended a focus on knowledge management -- impacts of 
cyberlearning explorations would be greater if insights more readily accumulated in 
sharable forms. Panelists noted that experts in information or library science could 
be untapped allies. Further, reviewing the Kind of interdisciplinary “convergence” that 
have proven useful in other novel research communities might shed new light on 
how the cyberlearning community could work more effectively together. 


Project Examples and Resources 


For project examples, resources, and links, visit: https://circlcenter.org/resources/reflections- 
report/emerging-trends/. From this site, you will be able to review project abstracts, read 
related publications, and view videos produced by the projects. 
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Conclusion: Recommendations for the Future 


As the reflections meeting came to a close, CIRCL asked panelists to reflect on the 
past, present, and future of cyberlearning. We received a dozen written reflections 
and also reviewed the recorded discussion. In so doing, several important 
recommendations arose and are presented below. 


Communicating Unique Characteristics of the Field 


As we observed earlier (see “History of Cyberlearning”), the community and its collection of 
research projects have taken shape gradually over about a decade. Although there is more of 
a sense of community now than when cyberlearning funding began, many participants, 
newcomers and community members are not yet clear about what makes this work 
distinctive: what's the difference between “digital learning” and ideas investigated in this 
smaller, more focused field? 


Digital learning is an expansive field of entrepreneurial, policy, and research interest. 
Generally speaking, much of the effort goes into figure out today’s digital resources can be 
applied effectively in the pursuit of learning. Research produces the knowledge to improve 
designs, support high quality implementations, and study efficacy. This research is important 
work and many NSF, U.S. Department of Education, philanthropic, and other sources of 
funding provide support for it. 


Cyberlearning research is born of a different problem—not how to apply today’s 
technologies, but getting ready for the learning technologies that will become prominent in 
the future. Underlying technologies and platforms are changing rapidly -- and yet research- 
based knowledge grows very slowly. Tomorrow's platforms are likely to be very different 
from those today, and so today’s knowledge may not readily extrapolate to new possibilities. 
Therefore, to be ready with research-based knowledge when the future arrives, the research 
about future learning needs to begin now. Cyberlearning most importantly has a focus on 
emerging technologies that are likely to become prominent 5-10 years from now. 


Following from this need, panelists celebrated a set of Key characteristics of the field: 


e Exploratory. Panelists strongly supported a focus on exploratory projects because we 
cannot definitively know what form the future will take, and thus we need to harness 
the imaginations of the designers. 

e Ambitious Mashups. Panelists strongly resonated with this concept (discussed 
earlier), which emphasizes explorations of unique combinations of emerging 
technologies. Explorations of mashups opens our eyes to divergent future 
possibilities. 

e Research and Design Iterations. Panelists valued cyberlearning as a field that iterates 
rigorously between design and research. The field builds prototypes in order to 
conduct research with them, and thereby to understand the theories, methods, and 
design patterns that will be important in the future. 

e Interdisciplinarity and Convergence. Panelists noted growth of practices that 
integrate multiple expertises, recommending that core teams include both computer 
science and learning sciences expertise. Of course, these two disciplines are just a 
beginning for many successful cyberlearning projects; other necessary disciplines 
need to engage and work convergently to answer shared research questions. 
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e Lifelong and Lifewide Learning. Many other research programs occur in silos 
according to the age, grade-level or special needs of the learner or are organized by 
the topic that students will learn. The field of cyberlearning projects notably traverses 
these boundaries. Multigenerational learning is explored. Researchers who worked 
with early learners, school-aged learners, university-aged learners and beyond 
exchange ideas. Boundaries between formal and informal learning are crossed. This is 
important. 


Going forward, panelists raised provocative remarks about where the field could further 
articulate its self-understanding and better argue for funding resources: 


1. What is the right type of shared infrastructure (what NSF terms “mid-level 
infrastructure”) that is needed to advance an exploratory community of 
researchers? 

2. How can the field better argue for its research-and-design activities as 
rigorous and capable of producing distinctive and much-needed outputs? 

3. How can the field better investigate “constellations” of technologies used over 
a variety of contexts in a person's lifetime and not only designs for particular 
learning experiences? 

4. How can the field sharpen its disciplined balance between unpacking the 
exciting possibilities of emerging technology and discovering how people 
learn equitably? 


Strengthening Reporting of Project Outcomes 


In the collection of projects we reviewed, clear examples were found of projects that pioneer 
ambitious mashups. Examples also develop important strands of learning theory, refine 
research methods, or bring valued new perspectives to addressing equity. While there is a 
sense of excitement and progress, panelists lamented that overall it is too hard to locate the 
insights and outcomes of cyberlearning projects. It is easier to find information on what will 
be explored than on what was learned through the exploration. 


Perspectives from panelists on this challenge overlapped around the following themes: 


e Clarity on Outputs. Broadly speaking, project outcomes include research findings 
and design insights, and include both “intellectual merit” and “broader impacts.” Yet 
there is a sense that the field is clearer about what kinds of things it explores than it is 
about what it produces. Funders, of course, want to know what outputs the funding 
leads to, so this is a critical issue for sustaining funding to the field. 

e Innovative Communication Formats. Panelists appreciated the efforts of CIRCL to 
advance novel dissemination formats, like primers, synthesis, and perspectives. 
Finding the right formats for documenting knowledge so it is “useful, intelligible, and 
relevant" to audiences is critical. Existing conferences and journals can be helpful, but 
novel venues and formats are judged by panelists to be very important. 

e Audiences. Panelists highlighted the broad audiences that care about and have much 
to learn from this field. One panelist suggested that shaping the public image of what 
learning looks like is very important. Others focused on audiences such as 
educational practitioners, industry and entrepreneurs, and policy-makers. 

e Themes. Panelists cautioned against assuming that themes like the previous NSF “Big 
Ideas” would continue into the future. They recognized that connecting to broader 
societal themes accelerate communication. For example, cyberlearning informs us 
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greatly about remote and online learning, which relates to the current pandemic. 
Also, themes of human-centered computing and social justice with Artificial 
Intelligence are on the rise. The field could re-think what broader societal themes 
best amplify its messages. 


Overall, CIRCL staff resonated with a recommendation to support the field in thinking about 
knowledge management, so that more outcomes and outputs of this research can be more 
readily available to multiple audiences. 


Enhancing the Equity Work 


Expert panelists advocated for maintaining a strong focus in cyberlearning projects on equity, 
including anti-racism and social justice. To the CIRCL team, equity has been a persistent and 
important feature of many projects. And yet, the equity-relevant characteristics of projects 
can seem diffuse. The field needs to more clearly define what its equity work in this field and 
how it expects to make progress. Further, stronger reporting of equity relevant outcomes is 
needed, whether those outcomes are design innovations, research findings, or new forms of 
participation in the process of investigating future learning. 


Panelists suggested a range of possible actions for the field in the future. These include: 


e Involving diverse participants in the work. Cyberlearning projects already have a 
strong history of women leaders as well as some history of leaders who a Black, 
Latinx, and other identities. Panelists suggested renewing efforts to cross-connect to 
HMCuUs and MSIs. Structurally, cyberlearning projects should be designed to include 
institutions and leaders who are not already privileged in their search for research 
funding. 

e Tackling pressing issues. Ethical issues regarding artificial intelligence and data 
analysis are worrisome to the field and to panelists; it is important to accelerate the 
field's progress on these issues. 

e Human-centered computing is another broad theme that implies participation of 
stakeholders in the world, and cyberlearning could lead by providing models of what 
it looks like to work on these issues together. 

e Openness in knowledge exchanges. Panelists also noted how sharing knowledge 
broadly is essential to creating an inclusive shared community that can collaborate on 
equity issues together. 
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Appendix 1: Findings from the 2020 Project Exit Survey 


We reviewed projects for their outcomes, which included reviewing answers to an exit survey 
question that asked projects to identify their contributions to each of computing and learning 
sciences. Based on this, we identified three classes of contributions that projects are making. 
We do not intend these to be levels nor categorizations of projects: all the classes of 
contribution are valuable and many projects pursue multiple classes of contribution 
simultaneously. We do think this simple framework may be useful in asking projects to report 
their outcomes in the future, so the contributions can be more readily captured and 
understood. We propose three classes of contribution: 


1. Advance. Projects often seek to use a particularly challenging situation or 
environment in learning with technology to drive a fundamental advance in 
their respective fields. In computing sciences, these come up often in 
collaborative learning situations—where sensing, detecting or recognizing 
learning-relevant features of human interaction is quite challenging and new 
techniques must be developed. In learning sciences, these often advance 
cutting edge theory on a concept such as embodied cognition, collaborative 
learning, or assessment—the technology-rich environment can be particularly 
fertile for advancing these areas of theory. 

2. Articulate. As previously mentioned, many projects explore the potential of a 
novel configuration of technologies and other learning resources, e.g., an 
ambitious mashup. The outcomes of such an exploration are often a better 
understanding of how two or more different promising approaches could 
interconnect, combine, or relate in service of learning—and how they can 
interoperate computationally. 

3. Advise. Projects also often develop guidelines, design principles, cases, 
exemplars, and other forms of documentation that can support a broader field 
to utilize what's known. Advising can also involve identifying problems that 
deserve more attention by the field of researchers, which can lead to a later 
Advance. 


Advance Examples: Examples: 


The project advances the e Detecting soeech moves e Refining theories of 
foundations of the field in naturally occurring embodied cognition 
by studying problem in collaborative discourse in an augmented 


learning Recognizing gestures as reality environment 


learners work with How whole body 
scientific models learning transfers 
more broadly in STEM 


Automatically sensing learning 


“rapport” among learners 
New forms of 
assessing learning 
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Articulate 


The project investigates a 
novel application or 
mashup of emerging 
advances in one or more 
fields, seeking to 
understand how to 
improve how support for 
learning 


Advise 


The project consolidates 
a range of insights that 
could be more broadly 
applied 


Examples: 


How data analysis 
techniques can be 
embedded in game-like 
learning environments 


How agent-based and 
system-dynamic models 
can fit together ina 


learning environment 


Examples: 


Recommendations on 
CS techniques for 
implementing universal 
design for learning 


How learning analytics 
could relate to 
assessment of 
competencies 


Examples: 


Articulating how 
critical theories and 
learning sciences 
theories relate 


Understanding 
whether haptics 
support young 
learner's embodied 
cognition 


Examples: 


Guidelines for 
pedagogy that Is both 
inclusive and 
innovative 


How advances In 
learning technologies 
could be applied in 
new settings, such as 
liberal arts colleges 


In reviewing many projects, our exit survey also identified several common drivers of 
innovation. These are contexts that occur in multiple cyberlearning projects and appear to be 
particularly fruitful for pursuing advances in cyberlearning that can span advances, 
articulations and advice classes of contribution. Some key drivers are listed below and further 
elaborated in our theme areas that follow. Our further theme areas also consider the 
portfolio more broadly than the exit survey (and thus is more inclusive than this list). 


e Understanding Collaborative Learning and Embodied Learning 

(beyond individual minds to include social learners and whole body learning) 
e Improving Assessment of Learning 

(beyond what is already measured well, for example, to new competencies) 
e Multimodal Teaching and Learning 

(beyond multiple representations to multiple forms of participation) 
e Learning Outside School or in Novel Space 

(beyond formal school structures and routines) 
e Advancing Capabilities of Artificial Intelligence 

(beyond tutoring or making recommendations) 
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